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INTRODUCTION 



Higher Education in the United States has evolved from a collection of 
basically private institutions serving essentially a local need and a somewhat 
elite clientele to a system of mass education serving a cross section of the 
population and addressing local, state, national, and even international 
needs. Along with this expansion ir size and breadth of activities, colleges 
and universities have become complex organizations. The administration of such 
complex organizations has required the adoption and development of specialized 
approaches to management that fit the unique mission, goals, and environment of 
an academic institution. One such approach is a type of evaluation known in 
higher education as "program review." This paper attempts to describe: (1) 
the development of program review in American higher education; (2) the present 
status of program review: and (3) to comment on prooable future developments. 

For the purposes of this paper, program review is defined as a special izec 
type of evaluation focusing on the evaluation of existing academic programs in 
American higher education. A "program" is defined as a sequence of courses of 
instruction leading to a degree. 

The paper begins with a description of the antecedents of program review, 
then provides an overview of its present status and concludes with some comment 
on its future. 



ANTECEDENTS 



It would be convenient if one could simply report that program review was 
invented in 1837 by Jeremiah P. Review, Professor of Higher Education at 
Harvard College, while working alone one evening in his Cambridge laboratory. 
Unfortunately, program review did not evolve in a single, easily identifiable 
beginning, nor does its beginning appear to have been the exclusive province 
of a single individual. Instead, tha antecedents of program review have 
been many and varied, with each providing differential contributions to what we 
today call "program review." 

The "roots" of program review go deep into the history of American higher 
education and are intermingled with major events in the development of higher 
education itself. Consequently, the antecedents described below are divided 
into five sections, each centered around a major development in U.S. higher 
education, namely, the development of (1) the concept of a "program"; (2) the 
emergence of the accreditation movement; (3) the emergence of the profession of 
educational evaluation; (4) the rise of the accountability movement; and (5) 
the development of sophisticated approaches to the management of higher educa- 
tion. E*ch of these developments in higher education provided the needed 
impetus for a simultaneous advancement in the development of program review. 

Development of Programs . When Henry Dunster set up the first course of 
study of the first English-American college, it resembled the academic program 
of Dunster's alma mater, Cambridge, as it existad in the early seventeenth 
century. It consisted of the hallowed "trivium" and "quadrivium" of classical 
antiquity. The central core of this kind of curriculum was the classical 
languages and literatures, as well as such subjects as Aramaic, Syriac, Hebrew, 
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ethics, politics, physics, mathematics, botany, and Oivinity. 1 This same basic 
type of classical curriculum remained in effect throughout the Colonial 
period. By 1765, however, the emphasis had shifted toward more attention to 
mathematics, natural sciences, English languages, and literature, with some 
modern foreign language instruction. This basic course of study was the only 
program offered leading to the Bachelor's degree for most early American 
colleges until the mid 1800s. 

Program reviews, or evaluations, of these "p^qrams," to the extent that 
they eristed at all, covered the entire breadth of the college curriculum. The 
primary type of evaluation was, in modern evaluative terminology, "outcome 
oriented, 11 since evaluations of the quality of instruction centered on the 
degree of proficiency in the classical subjects demonstrated by the students. 
This proficiency in turn was related directly to the faculty reward system, 
since a tutor's future often depended upon how well the instruction provided by 
the tu\,or resulted in students 1 demonstrated proficiency in the subjects 
covered. The job of the tutors was a difficult one and largely unrewarding, 
except to the degree that recognition of a good tutor led to employment in the 
ministry or some other permanent position. 2 

The curricular reform movement of the nineteenth century provided a new 
meaning to academic programs and perhaps the first important antecedent to 
program review. It evolved around the development of the "elective" system. 
An early version of this system was embodied in the "Virginia plan" advanced by 
Thomas Jefferson. In it, *he student was to have complete freedom of choice in 
choosing lectures between the offerings of the proposed University of 
Virginia's separate and distinct "schools." Once a student had chosen a field 
of specialization, no electives were permitted within a school, according to 



Brubacher and Rudy (p. 101). The success of the elective principle resulted in 
a broadening and expansion of the American college curriculum. According to 
Brubacher and Rudy, this eventually led to (1) the acceptance of a philosophy 
of the importance of all subjects; (2) the rise of scientific and utilitarian 
courses; (3) the development of subject matter special ization, with the 
attendant departmentalization of the curriculum; and (4) the seemingly endless 
proliferation o* courses. 3 The latter two aspects were also partly stimulated 
by the returning German-trained professors, who came to dominate many college 
faculties by the turn of the century. These professors frequently sought to 
carve out a domain for their own fields of specul ization. 

This "specialization' 1 was combined with an increasing preoccupation of the 
students with the utilitarian value of higher education to establish permanent- 
ly the concept of "academic programs." Just as the "lecture" replacec recita- 
tion as the primary method of instruction, a variety of programs had replaced 
the rigid classical curriculum. With the em* *gence of distinct programs of 
study, it theoretically became possible to have "program" review. Practically, 
however, the "review" aspect had still not yet matured. 

Beginning in the latter half of the 1800s, new forms of student assessment 
began to appear as a replacement of the Colonial pattern, where the student was 
essentially ungraded and unexamined. As Rudolph noted, "at the high-water mark 
of the classical college, grading and examining were poisoned by the recitation 
system and made somewhat ridiculous by the extent to which public oral examina- 
tions were gestures in public relations and therefore not designed to show up 
student deficiencies." 4 New forms of student evaluation eventually replaced 
the oral examinations with efforts at more precise measurement of proficiency. 
Harvard, for example, introduced the "blue book" for students to write course 
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exams to be graded by their instructors. b The development of program evalu- 
ation, however, awaited still ether historical developments. 

The Development of Accreditation . One of the first forms of programmatic 
evaluation emerged frc the institutions themselves r: an expression of concern 
over the absence of common standards and practices among all the new programs*. 
Out of a meeting of the National Association of State Universities in 
Wil 1 iamstown , Massachusetts, in 1906, accreditation emerged as a national 
phenomenon. The attendees, representatives of the four existing regional 
associations and the six-year-old College Entrance Examination Board, agreed 
to: 

* Recommend that the regional associations have their member 
colleges accept certificates from accredited schools in other 
regions; 

* Encourage the regional associations not yet doing so to organize 
"a college entrance certificate board or commission" for 
accrediting schools; 

* Propose the development of common definitions and standards; and 

* Establish a permanent commission "for the purpose of consider- 
ing, from time to time, entrance requirements and matters of 
mutual interest to colleges and preparatory schools."** 

This meeting was followed by a number of other meetings that resulted in 
recommended definitions (including the so-called Carnegie Unit), the admission 
testing program of the College Entrance Examination Board, and the nationaliz- 
ing of accreditation (first for the secondary level and later for colleges and 
universities). ^ Later in 1904, the Council on Medical Education of the 
American Medical Association established the first specialized or program 
accreditation. These developments had a profound change on American higher 
education, as Young, et al, have written: 

... a radically new concept suddenly appeared on the scene, was 
adopted and put into operation by colleges and universities in a 
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large area of the country, gained tne attention of a major profes- 
sional organization, and received the blessing and support of the 
leaders in the higher education community, an important foundation, 
and a key federal agency.** 

In the last seventy years or so, a number of important changes have 
occurred in accreditation which have, at least indirectly, contributed to the 
development of concepts related to program review. These changes include the 
fol lowing: 

* From a quantitative approach (expressed in specific require- 
ments) to a qualitative approach (based on more general stan- 
dards); 

* From an emphasis on making institutions more alike to recogniz- 
ing and encouraging institutional individuality; 

* From a system heavily dependent on external review to a system 
based more on self-evaluation and self-regulation; and 

* From an initial focus on judging an institution to a primary 
goal of encouraging and assisting an institu*">n to improve its 
educational quality.* 

The precise contribution of these developments to the emergence of 
program review is not easily traced. Some general relationships can be 
identified. The improvement of evaluative techniques, the general approach of 
having a self-study followed by visits of peers, the development of specialized 
and programmatic accreditation, the development of criteria to judge programs, 
the shift from quantitative to qualitative emphases have all had an impact on 
the way that program reviews have been conducted in jiany colleges and univer- 
sities. More recently, some accreditation bodies, such as the Southern 
Association, have actually taken steps to encourage institutions to conduct 
program review as a part of an effort to improve program quality. It appears, 
then, that the role of accreditation in the development of program review has 
been that of indirectly providing a related experience in program evaluation. 
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thi development of techniques for evaluation, and, in some instances, the 
actual encouragement of the program review itself. 

Emergence of the Profession of Evaluation . While examples of early forms 
of evaluation go deep into the history of education, such as the oral examina- 
tions given the students in Colonial days mentioned earlier, the development 
of the field of evaluation is much more recent. 

Evaluation as a profession grew as a result of various pressures, both 
internal and external. Often the impetus for evaluation came from outside 
academe, as was partly the case in the 1800s when the Massachusetts government 
proposed evaluation criteria in response to concerns about the vitality of 
Harvard's program. ^ Evaluation also grew as a reaction to many earlier evalu- 
ation techniques which concentrated on inputs and as a result of studies of 
curricula and education itself. Evaluative pioneers, su';h as E. L. Thorndike, 
who performed early examples of materials evaluation by analyzing the adequacy 
of arithmetic textbooks; G. Stanley Hall, who developed the use of question- 
naires and formative evaluation; and Ralph Tyler, who advocated a much broader 
range of student assessment and the development of cognitive variables, 
affective variables, and goals and objectives all made important contributions 
to the early development of the profession of evaluation. 

Early formal evaluations of academic programs are hard to find, as Popham 
has written: 

Educational historians who set out to recap the frequency with which 
formal educational evaluations have been conducted prior to the 
midpoint of this century are destined to do more hunting than 
finding. But starting around the 1950s, at least in the United 
States, developments occurred that led to a burgeoning interest in 
educational evaluation. 

The events that Popham refers to, however, occurred first with re?' ect to 
elementary and secondary education and not higher education, although there 
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were indirect implications for nigher education. These events included public 
dissatisfaction with the quality of public education, the infusion of federal 
dollars and concommitant requirements for evaluation, the shrinking financial 
support from local property taxes, the school consolidation movement, and 
attempts of educational improvement following the Soviet introduction of 
Sputnik. I* The public and its elected representatives sought evidence that 
public elementary and secondary education was doing a good job. They were, in 
effect, demanding that educational programs be evaluated. 

The field of program evaluation began to emerge slowly out of these 
various cries for greater accountability and improvement in education. As is 
typical of new academic fields, the initial contributions to the profession 
were made by persons trained in other related fields. Two landmark publica- 
tions were prepared Dy Michael Scriven,^ a philosopher, and R. E. Stake, ^ a 
psychometrician, in 1967. Eventually, the field of educational evaluation 
developed a literature of its own which chronicled a series of new develop- 
ments, such as: Adversary Model, Criterion-Referenced Measurement, Decision- 
Making Typology, Discrepancy Evaluation, Formative Evaluation, Goal-Free 
Evaluation, Item Sampling, Medical Model of Evaluation, Path Analysis, Second- 
ary Evaluation, Summative Evaluation, Trait-Treatment Interaction, Trans- 
actional Evaluation, and Unobtrusive Measures.^ 

In time, circumstances similar to those that had earlier occurred in 
elementary and secondary education took place with respect to higher education; 
and the new field of program evaluation began to be applied sporadically to 
academe. Many in higher education applauded and even helped conduct the 
evaluations in the public schools, but were not as enthusiastic about being 
evaluated themselves. These reactions were even stronger with regard to 
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degree program evaluation than the evaluation of other types of programs and 

special projects. The proponents of evaluation have grown over the years. 

Paul Dressel, for example, chided the academy with this statement: 

The jpporters of higher education must come to realize th^t even 
with the adaptation of new procedures, the increasing number of 
students and the increasing range of educational opportunity demanded 
will necessitate ever-increasing investment in higher education. The 
resolution of these issues is not likely to become evident so long as 
the discussion is marked by platitudes, criticisms, and recrimina- 
tions. What is needed is a research and evaluatory approach to the 
solution of the problems which we face. 16 

Evaluation has evolved as a profession with its own heros, terminology, 

literature, and folklore. As Popham's quote below indicates, . . \lso enjoys a 

greal dea 1 of popularity: 

Although perhaps not as widely sought as oeace, and not as revered as 
motherhood, educational evaluation is beginning to be given the same 
kind of unthinking adulation as these socially esteemed virtues. 

Accountability in Higher Education . The dramatic growth in higher 

education from the 1950s to the 1970s caused many to begin to question the 

quality of programs. The influx of federal and state dollars was often 

accompanied by evaluation requirements, such as the federal requirements for 

the annua, evaluation of vocational education programs in community colleges 

and technical institutes. A number of state executives and legislators became 

particularly frustrated over what they saw as uncontrolled growth, conflicting 

demands, and the increasing costs of providing mass higher education. One 

solution sought in a number of states was the creation of statewide boards. 

Beginning in the early 1900s a number of states established consolidated 

governing boards. These included Iowa (1909). Mississippi (1910), Idaho 

(1912), Kansas (1913), Oregon (192b;, Georgia (1931), and Arizona (1H5). In 

the latter half of the 1930s and in the 1960s a number of states adopted 

coordinating boards for higher education. By the 1970s these boards were 
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joined by o merged with the so-called "1202" commissions that were mandated as 
a condition for accepting certain federal higher education funds. * 8 Often the 
boards ^ re created to ii,_raase accountability and provide a rational approach 
to planning higher education. 

Howard Bowen has noted that: 

The idea of accountability is closely rented tn such concepts as 
program budgeting, cost-benefit effectiveness, and social indi- 
cators. It is the expression of a wish . . . felt by many public 
officials, donors, and tne general public, as well . . . that the 
increasing sums laid out for public services ought to be justified 
by reliable estimates ~ the outcomes. It reflects, in part, a 
failure of confidence in many of cur institutions and, in part, 
frustration over rapidly increasing costs, ^ 

Statewide coordinating and governinc boards generally have been given four 
specific types of responsibilities fcr wrylng out their role: (1) planning; 
(2) budgeting; (3) program review and approval; and (4) policy analysis. 20 All 
of these responsibilities had a relationship to program evaluation either 
directly or indi*- :t1 y . Planning required the assessment o f programmatic 
strengths and weaknesses, the programs were often a budgetary unit of analysis, 
and policy analysis frequently centered on programmatic issues. The new state- 
wide coordinating and governing boards frequently began to either require 
internal reviews or undertook external evaluations of their own. 21 

Statewide boards are just but one of many external groups that evaluate 
universities. Brown has indentified thirteen other external groups that also 
provide some form of external evaluation. 22 Many of these groups are also 
focusing in on programmatic concerns, but statewide boards and accreditation 
groups probably have had the greatest external impact on academic programs. 

Scientific Management . The growth and complexity and the calls for 
greater accountability in higher education also contributed to major develop- 
ments in its administration. 23 More and more, even the word ''administration" 
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itself began to be replaced by "management" terminology adopted from govern- 
ment, business, and industry. ^ The Ford Foundation, for example, funded a 
series of "management studies" in higher education. 25 M any outside of academe 
who felt colleges and universities should be more "businesslike." 

One highly heralded approach was the attempt to implement Planning, 
Programming, and Budgeting Systems (PPBS), a special type of systems analysis, 
in college an4 university management .26 This planning approach was adopted 
from similar approaches in business and government where it~ use appeared, at 
least to so;ne persons, to contribute to more effective management resources. 

In a PPBS approach, the university and its environment are considered to 
form a system. According to this approach, each activity or program is 
considered not only by itself, but also in relation to all the other activi- 
ties that make up a given university. The steps involved in establishing a 
PPBS system are closely related to program review and consist of: 

1. Establishing goals and objectives; 

2 Developing aHernate programs that will accomplish the same 
goals; 

3. Estimating resource requirements for each alternative; 

4. Estimating benefits to be gained from each program alternative. 

5. Developing an operating plan by selecting from among alterna- 
tives; 

6. Testing tne long-range fiscal implication of the plan; 

7. Computing the annual budget; 

*8. Evaluating the success of the program -- whether a program as 
perceived acc--rr.pl i shed its goals; 

9. Revising planning standards; and 

10. Repeating the cycle. ^ 
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The PPBS approach required the adoption ot highly ana ytic techniques in 
the assessment of all programs, such as "Cost-Benefit-Analysis, " which one 
proponent described as follows: 

Programs should be evaluated ... on the basis of their contribution 
to the attainment of t , e institution's goals. The maximum value 
should be obtained for the resources expended. Therefore, tie 
problem confronting the administr \tors is to find the best way to 
evaluate programs in light of goals so that the benefit of available 
resources 1s optimized. There is such a way — namely, Cost-Benefit- 
Analysis. CBA is a systematic means for evaluating any set of 
alternative programs. It involves inventorying the costs of ? 
program and the benefits of a program and comparing the net benefit 
of alternative programs on an objective basis. 28 

PPBS is but one example of the growth in scientific approaches to college 
and university management. It has not been a highly successful management tool 
in academe because of its having reliance on data, cost-benefit-analysis, and 
other techniques which are difficult to undertake. In terms of this discus- 
sion, PPBS and other types of systems analysis have contributed to the sophis- 
tication of programs reviewed and provided the groundwork for linking program 
review with other decision-making processes in colleges and universities. 

Summary o f Antecedents . The antecedents of program review have been many 
and varied, including the development of specialized programs of study, the 
emergence of regional and specialized accreditation and its efforts to assess 
th quality of programs, and the development of improved evaluation techniques 
stemming from emergence of the profession of evaluation. Many of these 
developments grew out of demands for greater accountability and concerns 
associated with the growth and complexity of higher education, sometimes 
resulting in the adoption of scientific management approaches to the adminis- 
tration of higher education. All of the factors combined over time to produce 
a new type of evaluation referred to as "program review. 11 

Proqram review's initial appearance came at two different ends rf the 
higher education spectrum — the vocational programs in the two-year institu- 
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tions and in the graduate, primarily doctoral, programs in the universities. 
The former, primarily due to the federal requirement that accompanied voca- 
tional education funds and the latter, in part to a concern about the quality 
of the programs due to their costs, orominence in the educational hierarchy, 
and their proliferation in numbers. This does not mean that there weren't 
program reviews taking place with respect to other degree levels, but that the 
major activities and the more systemat 4 - and formal efforts began with these 
two levels. 

Eventually, the techniques of program review were adopted for use at all 
degree levels as the need to be accountable and the desire for assurances of 
quality became widespread. Beginning in the 1960s, many of the newly developed 
statewide coordinating boards for higher education were created with the 
responsibility to approve new programs and review existing ones. 29 Like 
universities, the initial focus of other reviews was on the higher ~ost 
doctoral programs. Later, the techniques adopted for the doctoral programs 
were modified for use with other program levels. 

By the late 1970s the "typica"' internal program review consisted of a 
self-study conducted by the program under reviews' faculty and an assessment 
of the program by the program faculty or by peers of the program faculty. The 
latter was more likely in those instances where the reviews were a part of a 
larger institution-wide effort. 

Significant changes have occurred in the last decade in the nature and 
purpose of many reviews. The most important is probably the shift from reviews 
that were primarily undertaken as self-assessment of a program by its faculty 
for the purpose of improving it to a largely external evaluation undertaken by 
persons outside of the unit under review for the purpose of institution-wide 
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decision making concerning planning priorities and resource allocation (or, in 
evaluation jargon, from formative-type evaluations to summative evaluations). 
The demand for accountability requires that the evaluations be undertaken in a 
process that is objective — no longer is it acceptable for the h en to gua r d 
the chicken house. 

PRESENT 

There are six outstanding characteristics of program review today that are 
critical to any attempt to understand program review. First, it is now widely 
used in higher education; second, it has become more comprehensive; third, it 
is more systematic and formal; fourth, its purpose has shifted from formative 
to summative; fifth, it is Deing mre closely tied to other decision-making 
processes; and sixth, attitudes about program review are changing. 

Use of reviews is widespread . In a study reported in 1976, Glenny, 
et. al . , found that program review was the sole professional/administrative 
area in which more institutions anticipated increases in the future than 
experienced in them over the past six years. 30 for example, nearly one-third 
of Glenny's respondents expected an increase of over 5 percent in the number of 
. :af f employed in program review. While it is difficult to say with any 
precision if this has actually occurred, we do know that today approximately 82 
percent of colleges and universities (public and independent, two-year and 
four-year) conducted some kind of formal program review and another 2 percent 
have indicated that they conducted an informal review of programs. 3* The 
percentage of public institutions conducting such a review is considerauly 
higher than the percentage in the independent sector. This is due to a 
combination of factors, such as the relatively greater pressure on the public 
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institutions for accountability, the existence of statewide coordinating/ 
governing boards, the relatively greater administrative ana management efforts 
cf the public institutions, and the greater awareness o* program review in the 
public sector. 

The relatively greater percentage in the community college sector is due 
largely to its n?4d to conduct annual evaluations of vocational education as 
part of the federal vocational education requirements. The more comprehensive 
reviews, that is reviews covering all programs at an institution, occur most 
often at the public four-year institutions. The reviews are least likely to be 
found 'n the proprietary and specialized institutions. 

The vast majority of these reviews at all institutions have begun since 
1975. 32 

Reviews are more comprehensive . As noted earlier, program review in 
higher education was pretty much confined to the review of graduate programs 
in the four-year institutions and the vocational education programs at the two- 
year level. This has changed rather remarkably in recent years. Not only have 
the reviews been extended to undergraduate programs and academic programs not 
previously covered, but they have also been extended to cover continuing 
education and non-academic programs, as well. 

The reviews have also become more comprehensive in terms of the criteria 
used for review. Early reviews were often limited to several fairly simple 
criteria. Today the criteria for review are much more comprehensive and 
sophisl icated. The advent of computerized information systems and the avail- 
ability of personal computers now make possible the inclusion in the reviews of 
information formerly inaccessible and too time consuming. Also, several 
studies, such as the one undertaken by the Council of Graduate Schools in the 
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United States (CGS) and Education Testing Service (ETS), have identified a 
broad range of graduate program quality indicators that have been incorporated 
into instruments that can be used for program review. 33 

Reviews have become more systematic and formal . As noted above, some 82 
percent of all postsecondavy institutions report that they now have a formal 
process of review. There is a question whether all of these institutions 
really have a credible process of review, hot ?r, many institutions at least 
think they have such a formal process. There is no question, however, that 
program reviews have become more formal and systematic in the last decade. 
Where the reviews used to be conducted almost independently by various units 
based on more individualized approaches, they now are more centrally coordi- 
nated and are based on more standardized procedures. Specific time schedules 
for all campus reviews are more common and are frequently more closely adhered 
to than in the past. Evaluations of the review process itself are more common 
and have resulted in many major improvements to the reviews at all levels. 
Ljn^cn data systems now frequently link the various review efforts on campus. 

A primary purpose of the reviews is shifting . As the reviews have become 
more frequently used, more comprehensive and systematic, the purposes of the 
reviews on many campuses is shifting from reviews conducted primarily for 
program improvement to reviews conducted for purposes of program decision 
making, including possible termination. This is a significant change in purpose 
and has a profound impact not only on the reviews, but upon the institutional 
decision making processes, as well. 

Closer linkage to decision-making processes . Some of the reasons for the 
shifting purposes of the reviews include the need for retrenchment at many 
institutions and the integration of the reviews into other on-going decision- 
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end mcrementa, approaches to planning to strategic-tvo. „l ■ 34 

s i-u scrategic-type planning, 34 program 

re-ewhas become more cr.t.ca, to institut.ona, decision maMng. 

6U ° f Strne5,C " 9 today inCude an assessment of the 

strengths and weaknesses of academic programs. ,. Mny inst1tutjons , ^ 

essessment utilized for this purpose is essentia,,, prog 3 5 

spite of a„ these „„ k ,g. s to other d6c1sion . makf „ 5 , 

primary purpose of the reviews ,r ■ 

reviews at most institutions, especially four-year 
institutions, is still program improvement. 

PerSP ' rt f ^ ' - * Program review has become 
""^ e,fe " d «-* Panning and budgeting pro- 

cesses, if, perspective from the viewpoint of various constituencies has 

vice presidents, and board members, prog ew has risen to a new ,eve, of 

importance. Many presidents and board members, for exa TO ,e, can now guote th. 
^ "* ' '« ^e they previously bad only vague 

" 0t ' 0nS " ^ Pr ° 3 " ~ •» -t. faculty ,„ institutions with en- 
ding reviews nave mixed emotions about t ,, often ^ ^ 

PSrSOnS, " P * $ ' P™ reviewed „,„ if results 

were positive, tbey are positive, if not, they are not,. It has been t „ e 
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Figure 1 



CONTRIBUTIONS OF PROGRAM REVIEW TO STRATEGIC P'.ANNING PROCESSES 
IN COLLEGES AND UNIVERSITIES 
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author's experience that faculty at institutions without program review or 
those in institutions where expansion and improvement in reviews are being 
contemplated, increasingly have a negative connotation of program review and 
must be sold on its value. In the minds of many of these faculty, the reviews 
are being linked with program terminations and retrenchment efforts taking 
place in other institutions. This attitude is not without some justifi- 
cation. Unfortunately, this attitude ignores many of the positive aspects of 
program review, such as program improvement, better resources management, etc. 

The tremendous growth in interest and ^oicerns about program review has 
also resulted in myths about its adoption and use irt higher education. 

Webster defirns the word "myth," in part, as, "an ill-founded belief neld 
uncritically, especially by an interested group." This definition seems 
especially fitting in describing the "myths" •bout program review. The program 
review myths are frequen^v held uncritically by persons wishing to influence 
action (or non-action) regarding program review. Frequently, references are 
made to these myths as part of the rationale for justifying a particular 
action. Their use is not always made forthrigntly. 

Some of these myths are explored below: 

Hyth No, 1, You don't need program raview if your programs are ac- 
credited , A common reason noted by sdie faculty and administrators fo^ not 
undertaking program review is that their programs are already accredited by 
regional and sometimes specialized accrediting agencies. While both program 
review and accreditation are a type of evaluation and both can serve legitimate 
purposes, there is still a sufficient enough difference between the two to 
justify the use of both processes. To put it more positively, program review 
and accreditation are actually cornpl imentary. Several of the regional ac- 
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crediting bodies, for example, now encourage program reviews as an integral 
part of an institution's quality assurance program in preparation for accredi- 
tation. Reviews can also provide a continuity of program assessment between 
on-site accreditation visits. Accreditation, on the other hand, can bring the 
standards of a profession into consideration as a major part of a particular 
review. 

The rigor and requirements of the various specialized accrediting groups 
vary considerably, so there is also value in an institution having a common 
review process that crosses all academic programs and is based on a consistent 
list of criteria. The program review process then may go beyond the minimal 
standards frequercly required by the accrediting organizations and fill in the 
gaps where no recognized accreditation exists for particular programs. 

Myth No, 2. Identifying the criteria for review is the moit difficult 
step . There is some truth to this myth, but only because persons developing a 
program review process frequently confuse "criteria" with "indicators" of the 
criteria. Almost every comprehensive program review process uses the same 
basic criteria in some fashion. The^e common criteria are quality, cost, 
productivity, need/demand, and relation of the program to the mission of an 
institution, sometimes called "centrality." What is difficult and time 
consuming is the identification of indicators and related data Tor each 
criterion. For example, an indicator of graduate student quality could be a 
student's GRE scores; but frequently, some or all departments in a given 
siturtion cannot provide accurate data. 

The development of an information support system for the reviews may an 
extensive undertaking as current systems are reviewed and possibly restructured 
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and new systems are developed. This is where the real difficulty lies, 
however, and not in the identification of the criteria for review. 

Myth Mo. 3, Prog rail review is synonymous with program discontinuance . 
T here appears to be a growing tendency to equate program review with program 
discontinuance. Faculty members seem particularly susceptible to this inter- 
pretation, which is usually based on reading a sensational article describing 
how some institution or state board has eliminated "x" number of programs as a 
result of Us program review effort. Consequently, these faculty and others 
are frequently opposed to efforts aimed at developing a program review process 
on their campus. This association of program review with program discontinu- 
ance is unfortunate, because the actual use of reviews in higher education for 
this purpose is small; and even where it is used, its impact is limited. In 
addition, this myth obscures the many positive benefits that can occur from a 
wel 1 -developed program review process. 

While it is true that some program reviews do result in program discon- 
tinuance, the implications of a recent study by the author indicate that the 
vast majority of reviews are intended primarily, if not exclusively, for the 
purpose of program improvement.^ Whether or not programs are discontinued 
ought to depend on the purpose that the review is intended to serve. 

Even when the objectives of a program review process include possible 
program closure, this need not be a cause for alarm. The experience of most 
institutions that have discontinued programs as a result of program review has 
been that fewer than 3 percent (on average) of an institution's programs are 
actually discontinued and frequently some of these are "paper programs" that 
result in little impact beyond printing changes in the next version of the 
institution's catalog. The number of "real" programs actually discontinued 
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is very low, anc most of the institutions involved in such discontinuance have 
been exceedingly careful to minimize impact on faculty and students. Faculty 
retrai ni ng , phased reti rement . reass i gnment , and othor app roaches toward 
lessening the impact are frequently utilized. Students are often allowed to 
complete a program or are assisted in finding new alternatives. 

The structural outcomes of program review typically range from little or 
no change to program closure, with the majority of the actions taking place 
somewhere inbetween in the form of program modifications (i.e., resource 
allocation and re-allocation, staff reduction by attrition, changes in modes of 
delivery, mergers, etc.). 

The actual impact of wel 1 -developed reviews on the vast majority of an 
institution's faculty, contrary to the ^plications of this "myth, 11 can 
actually be very positive. Elimination or modification of a few programs (most 
of which were probably already held in low esteem on campus) can mean that 
there is a possibility of additional r purees being made available for the 
purpose of maintaining and improving other programs, especially those identi- 
fied *s having a higher priority. Most faculty feel positive about having 
"passed" the review by virtue of having the quality of the^r efforts sub- 
stantiated and having an opportunity to make a case for additional resources. 

Due to the negative context that this myth places on program review, ic 

also obscures many of the other positive contributions of program review to a 

college or university, such as: 

Improvement of academic program quality 
Identification of priorities among programs 
Increased assurance of consumer protection 

Heightened morale and optimism about ability to respond to changing 
conditions 

Increased attention to responsibilities to students 

Increased consideration of alternative ways to develop and deliver 
programs 
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Identification of furls for reallocation 
Improved planning and budgeting 

Myth No, 4, There is only one way to do program review . This myth 
is frequently held by both those who have a successful process and those who 
naively are searching for the one "true method," It is hard to tell someone 
who has been intimately involved in the development and implementation of a 
successful review process that that process, as good as it may seem to them, 
may not be the best process for another institution. Yet, the literature 
abounds with "show and tell" articles on program review, and there are plenty 
of true believers ready to initiate some >ne else's success story. 

The development of a successful program review process is not easy, so 
there is an eager audience searching for an easy way out by adopting review 
methodologies developed elsewhere. Unfortunately, those of us who have served 
as "repair" consultants on processes that have failed, all too often find that 
someone has tried to implement another institution's success story and it has 
failed (sometimes miserably). The fact is that the most successful program 
reviews are those that are developed indigenously to meet the needs, tradi- 
tions, and environment of a particular institution or system. Successful 
program reviews can occur in manv contexts or combinations thereof, with locus 
of control at many different levels and accomplished through many different 
types of processes. 

Myth No, 5. You can save money with program review , A frequently cited 
reason for undertaking program review is to "save money." While it is true 
that some institutions hava "saved money" as a result of program review, some 
cautions are in order. First, whether or not anything at all happens as a 
result of a given review depends upon a number of factors, not the least of 
which is the original objective of the review and the process developed to 
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iiplement th?t objective — they mu_w be consistent. As noted earlier, some 
reviews are 'Signed and implemented for the purpose of program improvement. 
Such reviews will seldom, if ever, result in a reduction in resources. In 
fact, reviews intended for the primary purpose of program improvement 3ually 
result in the identification of additional resource needs. Secondly, even when 
revi^ i have as an identified purpose the possibility of program discontinu- 
ance, instances of actual discontinuance are rare. As noted before, in some 
instances, the programs discontinued are "paper programs" that have no re- 
sources attached, so nothing can be saved. 

The "savings," if they c:cur at all, are the result of the elimination of 
actual rr* wee requirements of a given program, and primarily through the 
reduction of faculty through termination of employment, retirement, early 
retirement, reassignment, or other method. In addition, there are possi- 
bilities for "savings" as a result of future expenditures (obligations) that 
won't be spent and greater efficiencies in the institution as these funds can 
be reallocated to programs having a higher priority. For these reasons, it may 
be more useful to think in terms of "greater efficiencies" rather than "sav- 
ings." 

Finally, one should also be aware that program review itself has costs in 
money, time, and effort that need to be taken into censideraticn. Most 
institutions that have undertaken a cost-benefit analysis have concluded that 
the potential benefits ov the process outweigh these costs. 

Myth No, 6, Program review is expensive . The dollar costs of program 
review vary considerably from place to place. Sua: factors as the type of 
process utilized, the scope of tne re\ ews, and the staff time allocated to the 
process all influence the costs. The "typical" process that involves the use 
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of outside peer reviewers, and covers M programs at a given institution over 
a five to seven year period, is expensive (although even in this approach there 
is room for some economy). There are a number of alternative approaches to 
program review, however, that are less costly. Three examples are noted below: 

1. The use of a screening process to identify only programs chat 
would under go a more thorough review process, including outside 
peer review. This in-depth review would involve only a very few 
programs. The majority of the programs in institutions utiliz- 
ing this approach would be given only a "paper review, 11 based on 
standardized indicators of program viability. 

2. The use of standardized survey instruments to assess program 
quality, such as the Graduate Program Self-Assessment Surveys 
developed by the Educational Testing Service, as a major compo- 
nent of a review process. 

3. The use of more innovative approaches such as the "evaluation 
by discussion" technique, a five-phase process for program 
review utilizing internal persons on an expert panel. 37 

Myth No. 7. Proqraw rev 4 :*- will result in spectacular results . While 
this myth may sound a bit exaggerated, there are many persons who implement 
program review with the intention of quickly solving a number of major institu- 
tional problems and issues. Program review, if properly done, may do a lot of 
very valuable things for an institution, some of which are noted above. It 
will not, however, resolve everything that everyone expects, nor will it do 
much of anything quickly. In fact, many new administrative problems and issues 
may just begin as a result of the program review process. Such matter? as 
identifying programmatic priorities and qualitatively assessing programs often 
create r**M resource demands. A number of difficult allocation and reallocation 
decisions will need to be madu *s the program review results are integrated 
into budgeting and planning decisions. The review process itself may pr^e to 
be inadequate and ought to be evaluated for "fire tuning." Conseau^ntly, it is 
important that those implementing program review have realistic expectations. 
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Program review will most likely result in few major changes for most programs, 
and these changes may take years to implement. Nonetheless, the reviews .nay be 
important for an institution in order to remain viable and to distribute 
resou* es in a rational manner. 

In conclusion, several "myth. ' regarding the practice of program review 
have emerged in recent years. These myths frequently obscure the potential of 
program review to accomplish its full usefulness as a tool for institutional 
advancement. A recognition of these "myths" and proper development and 
implementation of program review processes can be helpful to institutions in 
efficiently providing quality academic programs. 



FUTURE 

Predicting the future of most anything is hazardous at best. There are 
some signs, however, that appear to shed some light on what may happen in the 
future, with respect to program review. The antecedents of program review 
provide a lesson on things to come. It was noted earlier that most of the 
major antecedents to program review occurred in response to circumstances and 
events taking place outside of academe. It is likely, therefore, that the 
future status of program review will be similarly impacted by external events. 

In many quarters, program ^eview has a status much like "Mom's apple 
pie." There is every indication that those who have provided the impetus for 
program review (i.e., the public and its elected and appointed representa- 
tives), as well as those in leadership positions in higher education, will 
continue to view program review in a favorable light. As Paul Dressel has 
written: 

Failure to engage systematically in evaluation in reaching the many 
decisions necessary in education means that decision by prejudice, by 

25 



o 23 

ERLC 



tradition, or by rationalization is paramount. Such oatterns of 
decision making are not consistent with the aims of education, 
particularly with those of higher education, which in our culture are 
based upon the assumption that informed judgments can and should be 
wiser judgments.^ 

The use of program review for administrative and external purposes may 
give rise to a conflict in purposes for which program review is being asked to 
serve. As noted earlier, program review has emerged from what was primarily an 
effort to improve programs to an effort to facilitate decision making with 
respect to planning and budgeting. These are obviously different purposes and 
require different or creative approaches. It would indeed be unfortunate if 
program review became solely an instrument for budgetary allocation rather than 
an instrument for improving the quality and effectiveness of an academic 
program. Those involved with program review in future yea^s will need to 
address this issue. 

In recent years, program review has become significantly more compre- 
hensive and sophisticated due to the availability of computers and our capacity 
to generate complex analyses of issues. The technologies in higher education 
are increasing at such a fast pace, it is difficult to comprehend what major 
changes could occur in just a few years. It is quite likely, however, that 
these technological capabilities will increase, resulting in new and even more 
sophisticated approaches to the review of academic programs. One sucn develop- 
ment could well be the use of computerized decision-making models, such as 
Interpretive S:ructural Modeling (ISM), which has been used in the business and 
governmental sectors. Through the use of such models, complex factors and 
large amounts of data can be analyzed and interpreted for use by decision 
makers. 
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Finally, it would seem unlikely that there will be very many institutions 
in a few years that will not be undergoing some k nd of program review. The 
trend in recent years is likely to continue or, into the future. Also, :here 
will continue to be (hopefully) improvements made in existing processes of 
program review. Many institutions have not yet availed themselves of the 
opportunities for significant improvement in their existing program review 
processes as a result of a comprehensive evaluation of the process itself. 
Truly innovative approaches are hard to find. As one who has done extensive 
consulting with colleges and universities regarding program review, there is a 
tremendous need for the evaluation of existing program review processes; and 
this evaluation component should become a standard operating procedure in every 
cycle of program reviews. A lot can be learned in such evaluations that could 
lead to positive improvements and new aporoaches. 

pr/ante 1/28/86 
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